Structure-inducing pre-training

Abstract Language model pre-training and the derived general-purpose methods have reshaped machine learning research. However, there remains considerable uncertainty regarding why improves performance of downstream tasks. This challenge is pronounced when using language in domains outside natural language. Here we investigate this problem by analysing how impose relational structure induced per-sample latent spaces—that is, what constraints do on distance or geometry between pre-trained embeddings samples. A comprehensive review reveals that question open, despite theoretical analyses showing importance understanding form structure. Based review, introduce a framework enables granular can be induced. We present analysis from first principles establish connection inductive bias fine-tuning performance. Empirical studies spanning three data modalities ten tasks confirm analyses, inform design novel consistent improvements over compelling suite methods.